Installing an Adapter

The adapter installation file contains an assets directory containing:

  • airflow_plugin_v1: Directory containing the plugin files required for Airflow 1.0

  • airflow_plugin_v2: Directory containing the plugin files required for Airflow 2.0

  • dags directory: Tidal wrapper DAGs for supporting special functionality of the adapter:

tidal_task_executor.py DAG supports triggering separate tasks in Airflow 1.0

tidal_task_exector_v2.py DAG supports triggering and canceling separate tasks in Airflow 2.0

Installing the adapter and a Tidal Bridge for Apache Airflow 1.0

Note: For Airflow 1.0 install a Tidal Bridge on the Airflow Server.

To install the Tidal Bridge for Airflow 1.0:

  1. Download the airflowplugin.zip plugin from the Tidal Support Center Downloads (Tidal Customer Portal > Support Center > Downloads) section.

  2. Unzip the airflowplugin.zip file. The unzipped file contains an assets directory with the airflow_plugin_v1, airflow_plugin_v2, and dags sub-directories.

  3. Copy the contents of the airflow_plugin_v1 directory to the Airflow plugin directory (airflow/plugins, for example).

  4. Copy the tidal_task_executor.py file to the Airflow DAG directory (airflow/dags, for example).

  5. Restart the Airflow Server and Airflow Scheduler.

Installing the adapter for Apache Airflow 2.0 in Google Cloud

Note: For Airflow 2.0 using Google Cloud install only the Tidal wrapper DAG.

Optional but recommended prerequisites

Airflow Configuration Overrides are required to connect the adapter. On the Airflow side configure

Note: When you first run Apache Airflow, the configuration properties are saved in the airflow.cfg file.

  • Required: Enables the connection

    webserver->expose_config = true

  • Optional: Allows spaces in the connection definition and job names

    scheduler->allowed_run_id_patters = ^[ A-Za-z0-9_.~:+-]+$

    Note: Pay attention to the space between "[" and "A" when configuring the settings. This space is important for the proper functioning of the configuration.

To install the adapter for Airflow 2.0 in Google Cloud:

  1. Download the airflowplugin.zip plugin from the Tidal Support Center Downloads (Tidal Customer Portal > Support Center > Downloads) section.

  2. Unzip the airflowplugin.zip file. The unzipped file contains an assets directory with the airflow_plugin_v1, airflow_plugin_v2 , and dags sub-directories.

  3. Copy the tidal_task_executor_v2.py file to the Airflow DAG directory (airflow/dags, for example).

  4. Restart the Airflow Server and Airflow Scheduler.

Installing the adapter and a Tidal Bridge for Apache Airflow 2.0 in an on-premise environment

Note: For Airflow 2.0 in an on-premise environment install a Tidal Bridge on the Airflow Server.

To install the Tidal Bridge for Airflow 2.0 in an on-premise environment:

  1. Download the airflowplugin.zip plugin from the Tidal Support Center Downloads (Tidal Customer Portal > Support Center > Downloads) section.

  2. Unzip the airflowplugin.zip file. The unzipped file contains an assets directory with the airflow_plugin_v1, airflow_plugin_v2 , and dags sub-directories.

  3. Copy the contents of the airflow_plugin_v2 directory to the Airflow plugin directory (airflow/plugins, for example).

  4. Copy the tidal_task_executor_v2.py file to the Airflow DAG directory (airflow/dags, for example).

  5. Restart the Airflow Server and Airflow Scheduler.